System and method for machine learning a confidence metric for machine translation

ABSTRACT

A machine translation system is trained to generate confidence scores indicative of a quality of a translation result. A source string is translated with a machine translator to generate a target string. Features indicative of translation operations performed are extracted from the machine translator. A trusted entity-assigned translation score is obtained and is indicative of a trusted entity-assigned translation quality of the translated string. A relationship between a subset of the extracted features and the trusted entity-assigned translation score is identified.

The present application is a divisional of and claims priority of U.S.patent application Ser. No. 10/309,950, filed Dec. 4, 2002, the contentof which is hereby incorporated by reference in its entirety.

BACKGROUND OF THE INVENTION

The present invention relates to machine translation. More specifically,the present invention relates to machine learning a confidence metricassociated with machine translation results.

Machine translation refers to the process of receiving an input stringin a source language and automatically generating an output string in atarget language. The output string will desirably be an accurate andfluent translation of the input string from the source language to thetarget language.

When translating a set of sentences using a machine translation system,the quality of the translations output by the machine translation systemtypically varies widely. Some sentences are translated accurately andfluently, others are translated adequately, but not necessarilyaccurately or fluently, and some (hopefully a small set) are translatedinto a translation result which is simply incomprehensible.

One primary application of a machine translation system is to aid humantranslators. In other words, as a human translator translates adocument, a component of helper software which is sometimes referred toas a translator's workbench attempts to minimize the human effortinvolved by consulting a database of past translations and suggestingtranslations that match the input string within a certain threshold. Inorder to perform properly, the translator's workbench must somehowdecide which of the translation hypotheses is most useful to a humantranslator. It has been found that if the translator's workbench choosesthe wrong translation hypotheses to display to the user, this mayactually waste more time than it saves because it confuses or misleadsthe human translator.

In prior systems, each individual rule used in the machine translationprocess was given a hand-coded score. The score was indicative of howwell each rule worked in the machine translation process. However, theindividual scoring was performed by doing a slow, hand-coded passthrough an entire machine translation system which is extremelyexpensive and subject to errors, in addition to being difficult tocustomize to different domains.

SUMMARY OF THE INVENTION

One aspect of the invention is implemented as two major phases: traininga machine translation system to produce confidence scores, and applyingthat process to produce a confidence metric during translation.

In the first phase, a machine translation system is trained to generateconfidence scores indicative of a quality of a translation result. Asource string is translated with a machine translator to generate atarget string. Features indicative of translation steps performed toobtain the target string are extracted from the machine translator. Atrusted entity-assigned translation score is obtained and is indicativeof a trusted entity-assigned translation quality of the target string. Acorrelation between a subset of the features and the trustedentity-assigned score is identified.

In one embodiment, a correlation coefficient is calculated. Acorrelation coefficient is associated with each of the extractedfeatures in the identified subset. The correlation coefficient isindicative of a correlation between the extracted feature and thetrusted entity-assigned score.

In the second phase, a translation result is generated that isindicative of a source string translated into a target string. Thetarget string is output along with a confidence metric that isindicative of an estimated trusted translation quality score.

In one embodiment, the confidence metric is calculated based on a set offeatures that is indicative of translation quality. The confidencemetric can be calculated using pre-calculated correlation coefficientsindicative of a correlation between each of the extracted features andthe trusted translation quality.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an illustrative environment in which thepresent invention can be used.

FIG. 2 is a more detailed block diagram of one embodiment of a system inaccordance with the present invention.

FIG. 3 is a flow diagram illustrating the operation of the system shownin FIG. 2.

FIG. 4 illustrates an extracted feature metric in accordance with oneembodiment of the present invention.

FIG. 5 is a block diagram of a machine translation system in accordancewith one embodiment of the present invention.

FIG. 6 illustrates a number of different extracted features inaccordance with one embodiment of the present invention.

FIG. 7 is a block diagram of a runtime machine translation system inaccordance with one embodiment of the present invention.

DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS

The present invention relates to generating a confidence metric in amachine translation system. The confidence metric is indicative of thequality of the translated output. However, prior to discussing thepresent invention in detail, a general description of the oneillustrative environment in which the present invention can be practicedwill be undertaken.

FIG. 1 illustrates an example of a suitable computing system environment100 on which the invention may be implemented. The computing systemenvironment 100 is only one example of a suitable computing environmentand is not intended to suggest any limitation as to the scope of use orfunctionality of the invention. Neither should the computing environment100 be interpreted as having any dependency or requirement relating toany one or combination of components illustrated in the exemplaryoperating environment 100.

The invention is operational with numerous other general purpose orspecial purpose computing system environments or configurations.Examples of well known computing systems, environments, and/orconfigurations that may be suitable for use with the invention include,but are not limited to, personal computers, server computers, hand-heldor laptop devices, multiprocessor systems, microprocessor-based systems,set top boxes, programmable consumer electronics, network PCs,minicomputers, mainframe computers, distributed computing environmentsthat include any of the above systems or devices, and the like.

The invention may be described in the general context ofcomputer-executable instructions, such as program modules, beingexecuted by a computer. Generally, program modules include routines,programs, objects, components, data structures, etc. that performparticular tasks or implement particular abstract data types. Theinvention may also be practiced in distributed computing environmentswhere tasks are performed by remote processing devices that are linkedthrough a communications network. In a distributed computingenvironment, program modules may be located in both local and remotecomputer storage media including memory storage devices.

With reference to FIG. 1, an exemplary system for implementing theinvention includes a general purpose computing device in the form of acomputer 110. Components of computer 110 may include, but are notlimited to, a processing unit 120, a system memory 130, and a system bus121 that couples various system components including the system memoryto the processing unit 120. The system bus 121 may be any of severaltypes of bus structures including a memory bus or memory controller, aperipheral bus, and a local bus using any of a variety of busarchitectures. By way of example, and not limitation, such architecturesinclude Industry Standard Architecture (ISA) bus, Micro ChannelArchitecture (MCA) bus, Enhanced ISA (EISA) bus, Video ElectronicsStandards Association (VESA) local bus, and Peripheral ComponentInterconnect (PCI) bus also known as Mezzanine bus.

Computer 110 typically includes a variety of computer readable media.Computer readable media can be any available media that can be accessedby computer 110 and includes both volatile and nonvolatile media,removable and non-removable media. By way of example, and notlimitation, computer readable media may comprise computer storage mediaand communication media. Computer storage media includes both volatileand nonvolatile, removable and non-removable media implemented in anymethod or technology for storage of information such as computerreadable instructions, data structures, program modules or other data.Computer storage media includes, but is not limited to, RAM, ROM,EEPROM, flash memory or other memory technology, CD-ROM, digitalversatile disks (DVD) or other optical disk storage, magnetic cassettes,magnetic tape, magnetic disk storage or other magnetic storage devices,or any other medium which can be used to store the desired informationand which can be accessed by computer 110. Communication media typicallyembodies computer readable instructions, data structures, programmodules or other data in a modulated data signal such as a carrier waveor other transport mechanism and includes any information deliverymedia. The term “modulated data signal” means a signal that has one ormore of its characteristics set or changed in such a manner as to encodeinformation in the signal. By way of example, and not limitation,communication media includes wired media such as a wired network ordirect-wired connection, and wireless media such as acoustic, RF,infrared and other wireless media. Combinations of any of the aboveshould also be included within the scope of computer readable media.

The system memory 130 includes computer storage media in the form ofvolatile and/or nonvolatile memory such as read only memory (ROM) 131and random access memory (RAM) 132. A basic input/output system 133(BIOS), containing the basic routines that help to transfer informationbetween elements within computer 110, such as during start-up, istypically stored in ROM 131. RAM 132 typically contains data and/orprogram modules that are immediately accessible to and/or presentlybeing operated on by processing unit 120. By way of example, and notlimitation, FIG. 1 illustrates operating system 134, applicationprograms 135, other program modules 136, and program data 137.

The computer 110 may also include other removable/non-removablevolatile/nonvolatile computer storage media. By way of example only,FIG. 1 illustrates a hard disk drive 141 that reads from or writes tonon-removable, nonvolatile magnetic media, a magnetic disk drive 151that reads from or writes to a removable, nonvolatile magnetic disk 152,and an optical disk drive 155 that reads from or writes to a removable,nonvolatile optical disk 156 such as a CD ROM or other optical media.Other removable/non-removable, volatile/nonvolatile computer storagemedia that can be used in the exemplary operating environment include,but are not limited to, magnetic tape cassettes, flash memory cards,digital versatile disks, digital video tape, solid state RAM, solidstate ROM, and the like. The hard disk drive 141 is typically connectedto the system bus 121 through a non-removable memory interface such asinterface 140, and magnetic disk drive 151 and optical disk drive 155are typically connected to the system bus 121 by a removable memoryinterface, such as interface 150.

The drives and their associated computer storage media discussed aboveand illustrated in FIG. 1, provide storage of computer readableinstructions, data structures, program modules and other data for thecomputer 110. In FIG. 1, for example, hard disk drive 141 is illustratedas storing operating system 144, application programs 145, other programmodules 146, and program data 147. Note that these components can eitherbe the same as or different from operating system 134, applicationprograms 135, other program modules 136, and program data 137. Operatingsystem 144, application programs 145, other program modules 146, andprogram data 147 are given different numbers here to illustrate that, ata minimum, they are different copies.

A user may enter commands and information into the computer 110 throughinput devices such as a keyboard 162, a microphone 163, and a pointingdevice 161, such as a mouse, trackball or touch pad. Other input devices(not shown) may include a joystick, game pad, satellite dish, scanner,or the like. These and other input devices are often connected to theprocessing unit 120 through a user input interface 160 that is coupledto the system bus, but may be connected by other interface and busstructures, such as a parallel port, game port or a universal serial bus(USB). A monitor 191 or other type of display device is also connectedto the system bus 121 via an interface, such as a video interface 190.In addition to the monitor, computers may also include other peripheraloutput devices such as speakers 197 and printer 196, which may beconnected through an output peripheral interface 195.

The computer 110 may operate in a networked environment using logicalconnections to one or more remote computers, such as a remote computer180. The remote computer 180 may be a personal computer, a hand-helddevice, a server, a router, a network PC, a peer device or other commonnetwork node, and typically includes many or all of the elementsdescribed above relative to the computer 110. The logical connectionsdepicted in FIG. 1 include a local area network (LAN) 171 and a widearea network (WAN) 173, but may also include other networks. Suchnetworking environments are commonplace in offices, enterprise-widecomputer networks, intranets and the Internet.

When used in a LAN networking environment, the computer 110 is connectedto the LAN 171 through a network interface or adapter 170. When used ina WAN networking environment, the computer 110 typically includes amodem 172 or other means for establishing communications over the WAN173, such as the Internet. The modem 172, which may be internal orexternal, may be connected to the system bus 121 via the user inputinterface 160, or other appropriate mechanism. In a networkedenvironment, program modules depicted relative to the computer 110, orportions thereof, may be stored in the remote memory storage device. Byway of example, and not limitation, FIG. 1 illustrates remoteapplication programs 185 as residing on remote computer 180. It will beappreciated that the network connections shown are exemplary and othermeans of establishing a communications link between the computers may beused.

FIG. 2 is a block diagram of a confidence metric training system thattrains a machine translation system to generate a confidence metricassociated with a translation output wherein the confidence metric isindicative of the quality of the translation output. System 200 includesmachine translator (or machine translation system) 202, evaluationcomponent 204, and relevant feature identifier and coefficient andconstant generator (generator) 206.

Machine translation system 202 can be any conventional machinetranslation system, but it is instrumented to produce a number ofstatistics, or features, that describe the translation process. Oneembodiment of machine translation system 202 is described in greaterdetail in Richardson et al., Overcoming the Customization BottleneckUsing Example-Based MT, ACL, COLING—2001. Evaluation component 204 isillustratively simply a component which allows human linguists (oranother trusted entity) to evaluate and assign a score to thetranslation results output by machine translation system 202. Generator206 can be any type of generator that identifies a correlation betweenthe features extracted from the machine translation system 202 duringthe machine translation process and the evaluation scores generated atevaluation component 204.

FIG. 3 is a flow diagram indicating the general operation of system 200.The operation of system 200 will now be described with respect to FIGS.2 and 3.

Machine translation system 202 first receives a plurality of testsentences 210. This is indicated by block 216 in FIG. 3. The sentencesare then translated by MT system 202. The operation of machinetranslation system 202 is described in greater detail in theabove-referenced paper. Briefly, in one embodiment, machine translationsystem 202 first parses test sentences 210 into a source languageintermediate structure that represents the actions, agents and objectsinvolved in the sentence, as well as relations between them. In oneembodiment, this source language intermediate structure is a sourcelanguage logical form. The process by which a logical form is generatedis known and is set out in U.S. Pat. No. 5,966,686 entitled Method andSystem for Computing Semantic Logical Forms from Syntax Trees, issuedOct. 12, 1999.

Then, by applying knowledge learned by parallel corpora and wordtranslation relationships, as well as translation dictionaries, thesource language intermediate structure is translated into acorresponding target language intermediate structure. Again, in oneembodiment, the target language intermediate structure is a targetlanguage logical form.

Finally, the target language intermediate structure is transformed intoa target language string. The target language string is represented bythe translation results 212 shown in FIG. 2.

From each of the translation phases in MT system 202, a set of featuresis extracted. In one embodiment, the set of features is a relativelylarge number of statistics that describe the translation process. Theset of features extracted from MT system 202 for each test sentence 210is indicated by block 214 in FIG. 2. Translating the test sentences andextracting the set of features is indicated by block 218 in FIG. 3.

The particular set of features 214 extracted from MT system 202 isdescribed in greater detail below. Translation results 212, along with ahuman-generated translation of test sentences 210, are provided toevaluation component 204. Evaluation component 204 assigns a trustedevaluation score to the translation results. The trusted score isindicative of the quality of the translation results associated witheach test sentence 210. The trusted evaluation score is assigned by atrusted entity which, in one illustrative embodiment, is a human and maybe a human linguist. The evaluation results for each sentence arerepresented by block 220 in FIG. 2, and the step of providing a trustedentity evaluation score of the translation results is illustrated atblock 222 in FIG. 3.

The evaluation results 220 can be generated in a wide variety of ways.For example, in one embodiment, each of the translation results 212,corresponding to each test sentence 210, along with a referencetranslation produced by a human, are presented to between 4 and 7independent human evaluators. The human evaluators assign a score ofbetween 1 and 4 to indicate the accuracy and fluency of the translation(e.g., 1 being completely unacceptable and 4 being an idealtranslation). The score from each evaluator can then be averaged andscaled to a range of 0-1. This represents the evaluation result for eachsentence indicated by block 220 in FIG. 2.

The set of features 214, along with evaluation results 220, are providedto generator 206. Generator 206 can take any number of forms, and itcomputes a function or correlation between the set of extracted features214 and the evaluation results 220. This function can be computed, forexample, using a multivariate linear regression solver, using decisiontrees, or using support vector machines (SVMs).

By way of example, assume that FIG. 4 represents a matrix of extractedfeatures 214 where the features are designated F1-FM and the sentencescorresponding to each feature vector are designated sentence 1-sentenceN. Also, assume that each sentence has a human evaluation score (ortrusted entity evaluation score) listed in the matrix as well.

Where generator 206 is a multivariate linear regression solver, it firstselects a feature from the feature matrix shown in FIG. 4, such asfeature F1. Generator 206 then determines how feature F1, across allsentences, correlates to the trusted evaluation score for each of thesentences. Generator 206 performs this step for each feature F1-FM andchooses the one that has the best correlation to the trusted evaluationscore. Generator 206 then calculates a correlation coefficientdescribing the correlation between the selected feature and the trustedevaluation score. The feature with the best correlation is placed on aselected feature list.

Generator 206 then attempts to add a second feature to its selectedfeature list. For instance, assume that during the first pass generator206 found that F3 had the highest correlation to the evaluation score.Thus, generator 206 will have selected feature F3 as the first relevantfeature to be used and placed it on the selected feature list. Nowgenerator 206 tires to add a second feature. Generator 206 selects oneof the remaining features for examination.

Assume for the sake of this example that generator 206 selects andexamines feature F1 for addition to the selected feature list. To dothis, generator 206 determines how closely the subset of features F3 andF1 correlate to the trusted evaluation score. Generator 206 performsthis step for each of the remaining features (i.e., it determines howclosely features F3 and F2 correlate to the trusted evaluation score, itdoes the same for features F3, F4 and F3, F5, up to F3, FM). Generator206 finds the best combination of features (i.e., the best additionalfeature to add to the selected feature list) and adds that feature tothe selected feature list. Generator 206 continues this process untilthe correlation between the features in the selected feature list andthe trusted evaluation score does not change by adding an additionalfeature from the set of extracted features 214 to the selected featurelist. The relevant set of features, along with the associatedcorrelation coefficients and constants (also collectively referred to asthe confidence metric parameters) are indicated by block 224 in FIG. 2,and the steps of processing the extracted features 214 and trustedevaluation scores 220 to obtain confidence metric parameters 224 isindicated by block 226 in FIG. 3.

Where generator 206 is comprised of some other technique, such asdecision trees or SVMs, the standard training process for thisembodiment is instead applied. The resultant subset of extractedfeatures selected by this training process along with the results of thetraining process itself—such as the actual decision tree (if usingdecision trees) or kernel parameters (if using SVMs)—then comprise theconfidence metric parameters 224.

Having found the subset of extracted features that are relevant tofinding a confidence metric, and having also identified a correspondencebetween these features and the trusted entity-assigned score, thecorrespondence parameters 224 are plugged into MT system 202 such thatit can now compute a confidence metric value for each translatedsentence during runtime. In order to do this, recall that MT system 202has already been instrumented to extract the set of features 214. MTsystem 202 simply extracts the relevant features 224 for each computedsentence and applies the confidence metric parameters 224 to compute theconfidence score. Where generator 206 is a multivariate linearregression solver, each extracted relevant feature is multiplied by thecoefficient derived by the multivariate regression solver, and the sumof those, together with the derived constant, produce the resultingconfidence metric score for the sentence being translated. Wheregenerator 206 trains decision trees, the decision tree is applied to therelevant features and the resulting probability is output as theconfidence score. Where generator 206 uses SVMs or some alternatemachine learning technique, the appropriate runtime usage of theconfidence metric parameters 224 is applied.

In one embodiment, the confidence metric value ranges between 0 and 1, 0meaning that the system has a very low confidence in the translation and1 meaning that the system has a very high confidence in the translation.These confidence metric values can be displayed to a translator who canchoose the translation result, modify it, or completely discard it basedon the confidence metric, or it can be used by a translator's workbenchto determine which translation results 212 are actually shown to thehuman translator. Of course, the confidence metrics can be used in anyother manner, as desired.

The extraction of the set of features 214, and the particular featuresextracted, will now be discussed in greater detail with respect to FIGS.5 and 6. It should be noted that attached as appendix A is a full set ofapproximately 200 extracted features which are extracted from MT system202 in accordance with one embodiment of the present invention. This isbut one exemplary list of extracted features, which are extracted fromMT system 202, and more, different, or fewer features can be extractedas well.

FIG. 5 is a more detailed block diagram of one embodiment of MT system202 instrumented to produce the extracted set of confidence metricfeatures 214 during translation of source sentence 210 into atranslation result (or target sentence) 212. The source sentence 210 isfirst provided to a parser 230 that parses source sentence 210 into theintermediate structure (such as a source language logical form). Thesource intermediate structure is indicated by block 232 in FIG. 5.

The parser 230 may also identify subcategories of nodes in the sourceintermediate structure. The count of nodes in a particular subcategorymay be emitted as a feature, and this subcategorization may be used bythe transfer component 240. The relationships between the nodes in thesource intermediate structure 232 may also be tracked in a correspondingway.

In one embodiment, the node categories include the following: Pronouns(such as “he”, “she” or “it”), Coindexed Node Copies (such as relativepronouns like “who” in “The man who is leaving lost his glove”), domainspecific multi-word units called Factoids (in the computer domain, theseinclude phrases such as “Microsoft SQL Server”), Coordinations (such as“and”, “or”, and “but”), and the remainder, which are simply calledOther Lemmas. In this same embodiment the relations are subdivided intoPrepositions (such as “with” and “under”) and Other Relationships.

During the parsing process, a number of confidence metric features 214are extracted. For example, it can be important to know whether parser230 was successful in generating a spanning parse—that is, a parse thatspans the entire source sentence 210—or whether the parser produced anon-spanning (or fitted) parse by conjoining parse trees for thesubsequences of the sentence that it was able to parse. Therefore,parser 230 outputs a value that simply indicates whether a spanningparse was found. It is believed that if the parse is fitted, thetranslation result will seldom be of good quality. Parser 230 alsooutputs, as a confidence metric feature 214, a value indicative of thesize of the source sentence. It is believed that very short sentencesare often difficult to translate successfully since the availablecontext to disambiguate word sentences and meanings is quite small. Onthe other hand, very long sentences are often quite difficult to parsecorrectly due to the inherent ambiguities of natural language. Thus, thesize of the source sentence (given, for example, simply as the number ofwords in the source sentence) is output by parser 230 as well.

During translation, source intermediate structures 232 are provided totransfer mapping identifier 234. Transfer mapping identifier 234 hasaccess to a lexical knowledge base 236, one embodiment of which isreferred to as the Mindnet knowledge base or database. The structure ofMindnet database 236 is described in greater detail in Richardson etal., MINDNET: Acquiring and Structuring Semantic Information From Text,Proceedings of COLING-ACL 98 (1998). Briefly, the Mindnet database 236is a repository holding transfer mappings from logical form alignment.Source and target language logical forms are aligned and transfermappings indicative of that alignment are placed in the Mindnet database236. However, any database that contains a mapping between source andlanguage intermediate linguistic structures can be used as well.

In any case, transfer mapping identifier 234 receives the sourceintermediate structure 232 and identifies transfer mappings in database236 that correspond to the source intermediate structure. In doing this,the source intermediate structure 232 is matched by transfer mappingidentifier 234 to source intermediate structures in transfer mappingdatabase 236. Multiple transfer mappings in the database 236 may matchportions of the source intermediate structure 232. Transfer mappingidentifier 234 can attempt to find the best match in database 236 basedon a variety of factors described in the above-referenced paper.

After a set of matching transfer mappings is found, identifier 234creates links on nodes of the source intermediate structure 232 tocopies of corresponding target intermediate structure segments retrievedfrom the mappings in database 236. Identifier 234 thus outputs a linkedintermediate structure indicated by 238 in FIG. 5. Linked intermediatestructure 238 is provided to transfer component 240.

Transfer component 240 can illustratively have access to a bilingualdictionary 242 and any other word association databases that are usedduring translation. Transfer component 240 receives the linkedintermediate structure 238 and creates a target intermediate structureby stitching together the linked intermediate structure fragments toobtain the target intermediate structure. This can be accomplished asdescribed in greater detail in the above-referenced Richardson paper.Briefly, it can be accomplished through a top down traversal of thelinked intermediate structure 238 in which the target intermediatestructure segments pointed to by the links in structure 238 are simplystitched together. In cases where no applicable transfer mapping isfound, the nodes of the source intermediate structure and theirtranslation are simply copied into the target intermediate structure.Also, single word translations may still be found from bilingualdictionary 242. Transfer component 240 thus outputs target intermediatestructure (e.g., target logical form) 244.

A large number of extracted features are extracted during the process oftranslating source intermediate structure 232 into target intermediatestructure 244. The multi-word mappings in database 236 are referred toas rules. Several features are extracted that relate to the applicationof those rules. For example, some features include the number of rulesused in translating the source intermediate structure 232 into thetarget intermediate structure 234, the average and maximum size of rulesused (larger rules tend to carry more context and/or are more likely tocapture idioms and domain expressions, hence leading to bettertranslations), the total frequency of all rules used in the translation,the average and weighted average of the rule frequency, and the averageand maximum rule alignment score. To explain the last feature a bitmore, each of the rules in database 236 may illustratively have aconfidence score associated with it. A confidence score illustrates theconfidence with which the rule was generated (i.e., how well the rulewas trained). Thus, when the rule is applied, that confidence score isextracted (or more particularly the average and maximum rule alignmentscore is extracted) as an extracted feature.

In addition, confidence metric features are extracted which indicatewhether the words in source sentence 210 were translated using database236, a domain translation dictionary (such as bilingual dictionary 242),using word association files learned from parallel corpora, or whetherthe words were left untranslated. This is described in greater detailwith respect to FIG. 6.

FIG. 6 is a matrix illustrating the source of the translation of wordsin the source sentence plotted against the types of words translated.For example, each of the words in the source sentence can be assigned toa word type, such as pronouns, prepositions, coordinations, factoids,coindexed node copies, etc. A number of those types of words areillustrated in FIG. 6 for the sake of example only.

Also, as discussed above, each word can be translated using a differentsource, such as the Mindnet database 236, bilingual dictionaries, wordassociation files, etc., or the words can be left untranslated. A numberof sources of translation are shown in FIG. 6 for the sake of exampleonly.

The number of occurrences of each type of word (pronoun, proposition,coordination, factoid, etc.) translated using each translation source(Mindnet, bilingual dictionary, word association file, etc.) can beplotted in the matrix.

Each of these numbers in the matrix can be an extracted confidencemetric feature 214, expressed both as an absolute count and as apercentage of the total. However, the totals can be extracted featuresas well, such as the total number of words in the sentence translatedusing each of the alternate sources of translation and the total numbersof the different types of words in the source sentence. The marginaltotals can be expressed both as absolute counts and as percentages.

After the target intermediate structure 244 is generated as shown inFIG. 5, it is provided to generation component 246. Generation component246 reads the target intermediate structure 244 and generates the targetsentence (or translation result) 212. Confidence metric features 214 canbe extracted by generation component 246 as well. For example, oneconfidence metric feature that is extracted is the perplexity of thetarget sentence 212. The perplexity of target sentence 212 is indicativeof a measure of the fluency of the generated text using a statisticallanguage model.

Having now identified relevant features and the correlation betweenthose relevant features and the trusted entity-assigned quality scores,a runtime system 400 will now be discussed with respect to FIG. 7.Runtime system 400 includes the instrumented MT system 202 and aconfidence computation component 402.

Source sentences 404 are received by MT system 202. MT system 202generates the translation results (translated sentences) 406. System 202also generates the confidence metric features that were identified bygenerator 206 as being relevant confidence metric features. This isindicated by block 408 in FIG. 7. Of course, it should be realized thatinstrumented MT system 202 can extract all of the features extractedduring the training process, or only the relevant subset of features.

In any case, the confidence metric features 408 are provided toconfidence computation component 402. Confidence computation component402 also has access to the confidence metric parameters (a list of therelevant subset of extracted features and the corresponding correlationcoefficients and constants associated with the relevant subset ofextracted features) 224. Confidence computation component 402 appliesthe confidence metric parameters 204 against the relevant confidencemetric features extracted by MT system 202 to generate the confidencescore associated with the source sentence 404 being translated. Thisconfidence score is added to the translation results 406 and theresultant output of the system includes the translated sentences alongwith the associated confidence scores as indicated by block 410 in FIG.7.

It should be noted that while the present discussion proceeds withrespect to a number of extracted features and relevant features,different or other features can be used as well. It may be desirable tohave a set of features that covers many aspects of the machinetranslation system, even if many are slight variations on one another.It can be difficult to determine which variations will have the bestpredictive power or correlation, especially when consideringcombinations of the different features. However, it is certainly notnecessary to have the exact feature set described herein, as even slightvariants may have comparable performance. It will also be noted thatdifferent subsets of extracted features may be relevant to thecorrelation, based on the particular domain being translated, the sourceand target languages, etc. Therefore, the present system need not belocked into any subset of relevant features.

It should also be noted that the techniques used in the presentinvention can vary widely as well. For example, although one techniquewas described for finding the relevant subset of extracted features,other techniques could be used as well. Instead of using a greedyalgorithm to find the relevant subset of extracted features, alldifferent combinations of extracted features could be tested althoughthis may undesirably increase machine learning time. Similarly, asmentioned above, completely different techniques can be used, such asdecision trees or support vector machines.

In any case, it can be seen that the present invention provides amachine translation system that outputs not only a translation result,but a translation result along with a confidence metric where theconfidence metric is indicative of a quality score that would beassigned by a trusted entity, such as a human. This is the case becausethe features used in generating the confidence metric are correlated totrusted entity-assigned quality scores during training.

Although the present invention has been described with reference toparticular embodiments, workers skilled in the art will recognize thatchanges may be made in form and detail without departing from the spiritand scope of the invention.

Appendix A: Example List of Confidence Metric Features.

-   Input size (number of source nodes) mappings-   Average mapping size-   Maximum match size-   Average align score-   Maximum align score target nodes-   Average link size target nodes divided by number of source nodes-   Fitted parse (1 or 0)-   Was there a root coordination (1 or 0)-   Was there coordination anywhere (1 or 0)-   Was there more than one parse (1 or 0)-   Total frequency in the training corpus of the mappings-   Average frequency of the mappings-   Weighted average (by size of match) of the mapping frequency-   Pronouns Total-   Pronouns translated from Mindnet-   Pronouns translated from Dictionary-   Pronouns translated from Rules-   Pronouns Lang-neutral-   Pronouns Untranslated-   Pronouns translated from Word-assoc-   Percent Pronouns translated from Mindnet-   Percent Pronouns translated from Dictionary-   Percent Pronouns translated from Rules-   Percent Pronouns Lang-neutral-   Percent Pronouns Untranslated-   Percent Pronouns translated from Word-assoc-   Pronouns Total divided by input size-   Pronouns translated from Mindnet divided by input size-   Pronouns translated from Dictionary divided by input size-   Pronouns translated from Rules divided by input size-   Pronouns Lang-neutral divided by input size-   Pronouns Untranslated divided by input size-   Pronouns translated from Word-assoc divided by input size-   Coindexed node copies Total-   Coindexed node copies translated from Mindnet-   Coindexed node copies translated from Dictionary-   Coindexed node copies translated from Rules-   Coindexed node copies Lang-neutral-   Coindexed node copies Untranslated-   Coindexed node copies translated from Word-assoc-   Percent Coindexed node copies translated from Mindnet-   Percent Coindexed node copies translated from Dictionary-   Percent Coindexed node copies translated from Rules-   Percent Coindexed node copies Lang-neutral-   Percent Coindexed node copies Untranslated-   Percent Coindexed node copies translated from Word-assoc-   Coindexed node copies Total divided by input size-   Coindexed node copies translated from Mindnet divided by input size-   Coindexed node copies translated from Dictionary divided by input    size-   Coindexed node copies translated from Rules divided by input size-   Coindexed node copies Lang-neutral divided by input size-   Coindexed node copies Untranslated divided by input size-   Coindexed node copies translated from Word-assoc divided by input    size-   Factoids as a unit Total-   Factoids as a unit translated from Mindnet-   Factoids as a unit translated from Dictionary-   Factoids as a unit translated from Rules-   Factoids as a unit Lang-neutral-   Factoids as a unit Untranslated-   Factoids as a unit translated from Word-assoc-   Percent Factoids as a unit translated from Mindnet-   Percent Factoids as a unit translated from Dictionary-   Percent Factoids as a unit translated from Rules-   Percent Factoids as a unit Lang-neutral-   Percent Factoids as a unit Untranslated-   Percent Factoids as a unit translated from Word-assoc-   Factoids as a unit Total divided by input size-   Factoids as a unit translated from Mindnet divided by input size-   Factoids as a unit translated from Dictionary divided by input size-   Factoids as a unit translated from Rules divided by input size-   Factoids as a unit Lang-neutral divided by input size-   Factoids as a unit Untranslated divided by input size-   Factoids as a unit translated from Word-assoc divided by input size-   Factoids via Facts and factgen rules Total-   Factoids via Facts and factgen rules translated from Mindnet-   Factoids via Facts and factgen rules translated from Dictionary-   Factoids via Facts and factgen rules translated from Rules-   Factoids via Facts and factgen rules Lang-neutral-   Factoids via Facts and factgen rules Untranslated-   Factoids via Facts and factgen rules translated from Word-assoc-   Percent Factoids via Facts and factgen rules translated from Mindnet-   Percent Factoids via Facts and factgen rules translated from    Dictionary-   Percent Factoids via Facts and factgen rules translated from Rules-   Percent Factoids via Facts and factgen rules Lang-neutral-   Percent Factoids via Facts and factgen rules Untranslated-   Percent Factoids via Facts and factgen rules translated from    Word-assoc-   Factoids via Facts and factgen rules Total divided by input size-   Factoids via Facts and factgen rules translated from Mindnet divided    by input size-   Factoids via Facts and factgen rules translated from Dictionary    divided by input size-   Factoids via Facts and factgen rules translated from Rules divided    by input size-   Factoids via Facts and factgen rules Lang-neutral divided by input    size-   Factoids via Facts and factgen rules Untranslated divided by input    size-   Factoids via Facts and factgen rules from Word-assoc divided by    input size-   Other Lemmas Total-   Other Lemmas translated from Mindnet-   Other Lemmas translated from Dictionary-   Other Lemmas translated from Rules-   Other Lemmas Lang-neutral-   Other Lemmas Untranslated-   Other Lemmas translated from Word-assoc percentage of Other Lemmas    translated from Mindnet-   Percent Other Lemmas translated from Dictionary-   Percent Other Lemmas translated from Rules-   Percent Other Lemmas Lang-neutral-   Percent Other Lemmas Untranslated-   Percent Other Lemmas translated from Word-assoc-   Other Lemmas Total divided by input size-   Other Lemmas translated from Mindnet divided by input size-   Other Lemmas translated from Dictionary divided by input size-   Other Lemmas translated from Rules divided by input size-   Other Lemmas Lang-neutral divided by input size-   Other Lemmas Untranslated divided by input size-   Other Lemmas translated from Word-assoc divided by input size-   Prepositions Total-   Prepositions translated from Mindnet-   Prepositions translated from Dictionary-   Prepositions translated from Rules-   Prepositions Lang-neutral-   Prepositions Untranslated-   Prepositions translated from Word-assoc-   Percent Prepositions translated from Mindnet-   Percent Prepositions translated from Dictionary-   Percent Prepositions translated from Rules-   Percent Prepositions Lang-neutral-   Percent Prepositions Untranslated-   Percent Prepositions translated from Word-assoc-   Prepositions Total divided by input size-   Prepositions translated from Mindnet divided by input size-   Prepositions translated from Dictionary divided by input size-   Prepositions translated from Rules divided by input size-   Prepositions Lang-neutral divided by input size-   Prepositions Untranslated divided by input size-   Prepositions translated from Word-assoc divided by input size-   Other relationships Total-   Other relationships translated from Mindnet-   Other relationships translated from Dictionary-   Other relationships translated from Rules-   Other relationships Lang-neutral-   Other relationships Untranslated-   Other relationships translated from Word-assoc-   Percent Other relationships translated from Mindnet-   Percent Other relationships translated from Dictionary-   Percent Other relationships translated from Rules-   Percent Other relationships Lang-neutral-   Percent Other relationships Untranslated-   Percent Other relationships translated from Word-assoc-   Other relationships Total divided by input size-   Other relationships translated from Mindnet divided by input size-   Other relationships translated from Dictionary divided by input size-   Other relationships translated from Rules divided by input size-   Other relationships Lang-neutral divided by input size-   Other relationships Untranslated divided by input size-   Other relationships translated from Word-assoc divided by input size-   Coord lemmas Total-   Coord lemmas translated from Mindnet-   Coord lemmas translated from Dictionary-   Coord lemmas translated from Rules-   Coord lemmas Lang-neutral-   Coord lemmas Untranslated-   Coord lemmas translated from Word-assoc-   Percent Coord lemmas translated from Mindnet-   Percent Coord lemmas translated from Dictionary-   Percent Coord lemmas translated from Rules-   Percent Coord lemmas Lang-neutral-   Percent Coord lemmas Untranslated-   Percent Coord lemmas translated from Word-assoc-   Coord lemmas Total divided by input size-   Coord lemmas translated from Mindnet divided by input size-   Coord lemmas translated from Dictionary divided by input size-   Coord lemmas translated from Rules divided by input size-   Coord lemmas Lang-neutral divided by input size-   Coord lemmas Untranslated divided by input size-   Coord lemmas translated from Word-assoc divided by input size-   Total number of words translated from Mindnet-   Total number of words translated from Dictionary-   Total number of words translated from Rules-   Total number of words Lang-neutral-   Total number of words Untranslated-   Total number of words translated from Word-assoc-   Total number of words translated from Mindnet divided by input size-   Total number of words translated from Dictionary divided by input    size-   Total number of words translated from Rules divided by input size-   Total number of words Lang-neutral divided by input size-   Total number of words Untranslated divided by input size-   Total number of words translated from Word-assoc divided by input    size

The Following Set are Computed After Factoids are Added to Other Lemmasand Language Neutral Numbers are Ignored

-   Pronouns Total-   Pronouns translated from Mindnet-   Pronouns translated from Dictionary-   Pronouns translated from Rules-   Pronouns Lang-neutral-   Pronouns Untranslated-   Pronouns translated from Word-assoc-   Percent Pronouns translated from Mindnet-   Percent Pronouns translated from Dictionary-   Percent Pronouns translated from Rules-   Percent Pronouns Lang-neutral-   Percent Pronouns Untranslated-   Percent Pronouns translated from Word-assoc-   Pronouns Total divided by input size-   Pronouns translated from Mindnet divided by input size-   Pronouns translated from Dictionary divided by input size-   Pronouns translated from Rules divided by input size-   Pronouns Lang-neutral divided by input size-   Pronouns Untranslated divided by input size-   Pronouns translated from Word-assoc divided by input size-   Other Lemmas Total-   Other Lemmas translated from Mindnet-   Other Lemmas translated from Dictionary-   Other Lemmas translated from Rules-   Other Lemmas Lang-neutral-   Other Lemmas Untranslated-   Other Lemmas translated from Word-assoc-   Percent Other Lemmas translated from Mindnet-   Percent Other Lemmas translated from Dictionary-   Percent Other Lemmas translated from Rules-   Percent Other Lemmas Lang-neutral-   Percent Other Lemmas Untranslated-   Percent Other Lemmas translated from Word-assoc-   Other Lemmas Total divided by input size-   Other Lemmas translated from Mindnet divided by input size-   Other Lemmas translated from Dictionary divided by input size-   Other Lemmas translated from Rules divided by input size-   Other Lemmas Lang-neutral divided by input size-   Other Lemmas Untranslated divided by input size-   Other Lemmas translated from Word-assoc divided by input size    Appendix B: The Feature Set Selected by One Run of the Multivariate    Linear Regression Solver.-   Average mapping size-   Fitted parse-   Input size-   Maximum align score-   Maximum match size-   Number of Factoids via Facts and factgen rules Total-   Number of Pronouns translated from Mindnet-   Number of Pronouns Total divided by input size-   Output logical form size-   Other Lemmas Untranslated

1. A method of training a machine translation system to generateconfidence scores indicative of a quality of a translation result,comprising: translating a source string with a machine translator togenerate a target string; extracting features from the machinetranslator, indicative of performance of translation steps in themachine translator; obtaining a trusted entity-assigned translationscore indicative of a trusted entity-assigned translation quality of thetarget string; and identifying a relationship between a subset of theextracted features and the trusted entity-assigned translation score. 2.The method of claim 1 wherein translating a source string comprises:translating a plurality of source strings to generate a plurality ofcorresponding target strings.
 3. The method of claim 2 whereinextracting features comprises: extracting a plurality of featuresassociated with each of the plurality of target strings.
 4. The methodof claim 1 wherein translating includes: parsing the source string intoa source intermediate linguistic structure indicative of a meaning ofthe source string and wherein extracting features comprises: extractingone or more features indicative of a quality of parsing.
 5. The methodof claim 4 wherein extracting one or more features indicative of aquality of parsing comprises: extracting a feature indicative of whetherparsing generated a fitted parse as the intermediate structure.
 6. Themethod of claim 1 wherein extracting features comprises: extracting afeature indicative of a size of the source string.
 7. The method ofclaim 6 wherein translating includes translating the source intermediatelinguistic structure to a target intermediate linguistic structure andwherein extracting features comprises: extracting one or more featuresindicative of a quality of translating the source intermediatelinguistic structure into the target intermediate linguistic structure.8. The method of claim 7 wherein translating the source intermediatelinguistic structure comprises: identifying mappings, in a mappingdatabase, that map portions of the source intermediate linguisticstructure to portions of the target intermediate linguistic structure.9. The method of claim 8 wherein extracting one or more featuresindicative of a quality of translating the source intermediatelinguistic structure comprises: extracting a feature indicative of anumber of identified mappings.
 10. The method of claim 8 whereinextracting one or more features indicative of a quality of translatingthe source intermediate linguistic structure comprises: extracting afeature indicative of a size of identified mappings.
 11. The method ofclaim 8 wherein extracting one or more features indicative of a qualityof translating the source intermediate linguistic structure comprises:extracting a feature indicative of a frequency with which mappings whereidentified.
 12. The method of claim 8 wherein each mapping has anassociated confidence score and wherein extracting one or more featuresindicative of a quality of translating the source intermediatelinguistic structure comprises: extracting a feature indicative of theconfidence scores associated with the identified mappings.
 13. Themethod of claim 8 wherein translating the source intermediate linguisticstructure comprises: translating portions of the source intermediatelinguistic structure using another translation device other than themapping database.
 14. The method of claim 1 wherein extracting featurescomprises: extracting a feature indicative of the translation deviceused.
 15. The method of claim 14 wherein extracting features comprises:extracting a feature indicative of an amount of the source stringtranslated with another translation device.
 16. The method of claim 14wherein extracting features comprises: extracting a feature indicativeof a type of words translated with another translation device.
 17. Themethod of claim 13 wherein some words in the source intermediatelinguistic structure remain untranslated and wherein extracting one ormore features indicative of a quality of translating the sourceintermediate linguistic structure comprises: extracting a featureindicative of untranslated words in the source intermediate linguisticstructure.
 18. The method of claim 1 wherein extracting featurescomprises: calculating a perplexity of the target string with astatistical language model.
 19. A confidence metric training system fortraining a machine translation system to translate a source string intoa target string and provide a confidence metric associated with thetarget string, the confidence metric training system comprising: amachine translator configured to receive the source string and translatethe source string into the target string and to extract a plurality offeatures indicative of translation steps performed in the translation;and a relevant feature identifier configured to receive the extractedfeatures and a trusted entity-assigned (TEA) score indicative of atrusted entity determination of a quality of the translation of thesource string into the target sting, and to identify a relationshipbetween a subset of the extracted features and the TEA scores.
 20. Amethod of generating a translation result indicative of a source stringtranslated into a target string, comprising: translating the sourcestring into the target string using a machine translator; and outputtingthe target string and a confidence metric, the confidence metric beingindicative of an estimated trusted translation quality score.